Skip to content

internal/loopir, internal/rewrite: vectorize stores at a loop-invariant offset - #2

Closed
thesyncim wants to merge 5 commits into
masterfrom
offset-store
Closed

thesyncim wants to merge 5 commits into
masterfrom
offset-store

Conversation

@thesyncim

@thesyncim thesyncim commented Oct 31, 2026 •

Copy link
Copy Markdown
Owner

Loads can already be at i+e, e+i or i-e for a loop-invariant e. This allows the same for stores, which covers row-wise code like:

for j := 0; j < n; j++ { b[i*ldb+j] = a[i*lda+j] }        // gonum dlacpy
for j := 0; j < k; j++ { c[i*ldc+j] -= work[i*ldwork+j] } // gonum dlarfb
for x := range w { line[off+x] = v }

Every store to a given slice has to use the same offset, so each iteration still writes its own element. Two different offsets are rejected with a new reason. For loads of a stored slice, the distance is now measured from the store offset instead of from i. The same offset means the same element. A constant positive distance in a forward loop is loaded before the chunk's stores, as today. A negative distance, a non-constant one, or a loop that counts down is rejected. The store is capped at limit+offset, and the length check and the overlap window include the offset. The if-select and copy-builtin paths take the offset too.

With loopvec -json -methods (I read every added loop), std goes from 34 to 35 vectorized loops (an image/jpeg fill), gonum from 161 to 177 (dlacpy, dlaset, dlascl and dlarfb rows, plus some testlapack helpers), and goav1 from 81 to 86. TSVC s423 and s424 now vectorize and match their goldens. The generated code for loops that were already vectorized doesn't change.

offset_store.txt runs 9 shapes over 14 sizes, plus short slices that must panic and aliased views that must fall back. offset_store_no_rewrite.txt pins the rejected cases.

Heads-up: this conflicts with #3, and git merges the two without a conflict into wrong code, because destination() here doesn't know about #3's windows. Whichever one lands second needs destination to defer to the window when there is one. I have that version ready on thesyncim:offset-store-on-windows and can push it here once #3 is in.

The check for stored indices below zero reuses the one from #1.

ubuntu-latest is now ubuntu-26.04, which has no qemu-user-static
package, so every run fails at the install step and skips the arm64
tsvc run, the golden check, fmt and lint. qemu-user ships the static
emulators there; link qemu-aarch64-static to it when the name is
missing, as the Makefile runs that name.
Go tip writes export data in iexport version 4, which encodes float
constants exactly. x/tools v0.51.0, and the v0.52.0 release, read only
up to version 3, so on tip go/packages fails for every package:

  cannot import "math" (unstable iexport format version 4, just rebuild
  compiler and std library), export data is newer version - update tool

and every script that runs loopvec fails in the tip job. x/tools master
reads version 4 (golang/tools@966056e2637c). Update to it until a release
includes it. The pinned toolchain is unaffected.
go1.27.3 is the latest 1.27 release. Its simd package is the same as
go1.27.1's, so the capability table only changes in its header, and
the -methods crash (golang/go#80657) is still there, as
flag_methods_ice.txt checks. go.mod keeps go 1.27.1: that is the first
release whose simd package loopvec's output compiles against
(go1.27.0's ShiftAllRight takes a uint8).
A read at a negative constant offset in a loop with a constant limit, such
as dst[i] = a[i-20] + 1 in a loop that stops before 16, was rewritten into
code that does not compile: the length check of the vector loop is
_ = a[16-21], a negative constant index. -split wrote that file, and the
toolexec wrapper fell back to the original package.

Such a loop panics as soon as it runs, as does one whose first index is
a negative constant (a[i-1] in a loop from 0). Both are now left alone
with their own reason, the way a constant limit of zero or less already
is. A read whose indexes can all be zero or more (a[i-1] from 1) is
still rewritten.

The checks read the start and limit as text, so a named constant or a
constant expression (const s0 = 0; for i := s0; ...) slipped past them.
A constant start or limit is now spelled as its value, which also lets
the short-loop rule see a named constant trip count: goav1's
for x := range restorationExtraHorz, three iterations, is no longer
rewritten.

The vectorized set of std and gonum does not change.
…nt offset

A loop that stores at an offset from the index, such as a row of a 2-D
slice (b[i*ldb+j] = a[i*lda+j], c[row+j] += alpha*x[j]) or a fill that
starts part way into a slice (line[off+x] = v), was rejected: loads could
be at an offset, but every store had to be at exactly the loop index.

A store may now be at the same offsets a load may: i+e, e+i and i-e for a
loop-invariant integer e. Every store to one slice must be at the same
offset, so each iteration still writes its own element; two offsets are
rejected with their own reason. A load of the stored slice at that offset
is the same element, ordered after the stores before it as at the index.
The dependence test for a load at another offset measures the distance
from the store offset instead of from the index, so a constant positive
distance in a forward loop is still loaded before any store of the chunk,
and a negative or variable distance is rejected as before. An if that
selects between stores, and the copy builtin, take the offset too.

The emitted store is capped at the limit plus the offset, so the vector
loop never writes an element the scalar loop does not, and the length
check and the overlap window cover the elements stored. A constant offset
that puts every stored index below zero is left alone, as for a load.

The vectorized set gains one loop in std (an image/jpeg fill), sixteen in
gonum (dlacpy, dlaset, dlascl and dlarfb rows, and testlapack helpers)
and five in github.com/thesyncim/goav1; TSVC s423 and s424 now vectorize
and match their goldens.
@thesyncim

Copy link
Copy Markdown
Owner Author

Superseded by #14, the same change stacked on #11 so it works with the window loops from #3.

@thesyncim thesyncim closed this Oct 31, 2026
mauri870 added a commit that referenced this pull request Nov 3, 2026
…ro (#2)

loopvec writes a `_simd.go` that doesn't compile for a loop like this:

```go
for i := 0; i < 16; i++ {
	dst[i] = a[i-20] + 1
}
```

The length check comes out as `_ = a[16-21]`:

```
invalid argument: index 16 - 21 (constant -5 of type int) must not be negative
```

toolexec falls back, but `-split` and `-w` leave the broken file behind.

A loop like that panics as soon as it runs, and so does one whose first
index is a negative constant (`a[i-1]` starting at 0). Both are now
rejected with their own reason, the same way a constant limit of zero or
less already is. `a[i-1]` starting at 1 is still rewritten.

The checks parsed the start and limit as text, so named constants and
constant expressions (`const s0 = 0; for i := s0; ...`, `for i := 1 - 1;
...`) slipped through. Constant bounds are now written out as their
value. A side effect is that the short-loop rule now sees named constant
trip counts too: in [goav1](https://github.com/thesyncim/goav1), a
3-iteration `for x := range restorationExtraHorz` is no longer
vectorized. The vectorized set of std and gonum doesn't change.

Co-authored-by: Mauri de Souza Meneguzzo <mauri870@gmail.com>
mauri870 added a commit that referenced this pull request Nov 3, 2026
)

Follow-up to #2.

#2 started replacing every constant start or limit with its value, so
that the checks parsing `Trip.Start`/`Trip.Limit` as text would also see
named constants. The emitted file is compiled for every build, though,
and a named constant can have a different value in another one. With
`const width = 32` for amd64 and `64` for arm64, `loopvec -split` on
amd64 hard-coded `32` into the vector loop of `for i := 0; i < width;
i++`, while the scalar fallback kept `width`. The arm64 build would then
vectorize only half the elements. `plusOne` and `i > low` loops had
always done the same for `<=` limits and loops that count down.

What changes:

- **Constant names are kept.** `Trip` carries the values of a constant
start and limit. The checks read those values instead of parsing text:
the negative-index check, the short-loop rule and the copy check. A
constant is written as its value only when it's built from literals
alone (`16`, `1 - 1`). A constant that uses a name keeps the name. `i <=
width-1` comes out as `width`.
- **Dead code removed.** `negativeSource` could no longer return true
after #2.
- **The negative-index check can't overflow.** It negates the offset
instead of adding it to the bound. It also runs in the same pass over
the loop body as the stored-offset checks.
- **Bounds pushed past the largest int are rejected.** A positive offset
that pushes a constant start or limit past the largest int is rejected
as that bound. `for i := 0; i < math.MaxInt64; i++ { dst[i] = a[i+1] }`
used to produce an overlap window of `math.MaxInt64 + 1`, which doesn't
compile.
- **Tests:**
- `offset_negative_constant.txt` now checks the rewritten source and
runs the loops under both builds.
- The rejected loops moved to `offset_negative_constant_no_rewrite.txt`,
which adds a loop counting down to 0, a range over a slice, and the two
loops pushed past the largest int.
- `bound_arch_constant.txt` covers the GOARCH case for `<`, `<=`,
counting down and `>`.
- **AGENTS.md** documents the new rejections and the rule for keeping
names.

Tested locally:

- `bound_arch_constant.txt` fails against #2's `internal/loopir`, which
writes `_ = dst[32-1]` where the test expects `_ = dst[width-1]`.
- `make test` and `make test-qemu-arm64` pass.
- `make test-update` and `make fix` leave no diff, and `make lint`
reports 0 issues.
- `loopvec -json` gives identical output before and after over std (1663
loops) and gonum (3252 loops).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant